The U.S. weighed curbs on software-related exports to China, responding to Beijing’s rare earth export controls, ahead of a critical round of negotiations.