Let λ ∈ ℝ, a⃗=λ î+2 ĵ-3 k̂, b⃗=î-λ ĵ+2 k̂.
If ((a⃗+b⃗) ×(a⃗ × b⃗)) ×(a⃗-b⃗)=8 î-40 ĵ-24 k̂,
then |λ(a⃗+b⃗) ×(a⃗-b⃗)|² is equal to :

Solution

Maths sir · 5 steps

Call a⃗+b⃗ and a⃗-b⃗ by new names, and that scary triple product shrinks to one line.

1. Rename. Let p⃗=a⃗+b⃗ and q⃗=a⃗-b⃗. Expanding gives p⃗×q⃗=-a⃗×b⃗+b⃗×a⃗=-2 a⃗×b⃗, so a⃗×b⃗=-1/2 p⃗×q⃗.
2. Collapse the product. By the triple product rule, p⃗×(p⃗×q⃗)=(p⃗·q⃗)p⃗-|p⃗|²q⃗. Crossing this with q⃗ removes the second term, because q⃗×q⃗=0⃗.

   (p⃗×(a⃗×b⃗))×q⃗=-1/2 (p⃗·q⃗) (p⃗×q⃗)

3. Use the dot product. p⃗·q⃗=|a⃗|²-|b⃗|²=(λ²+13)-(λ²+5)=8, and λ drops out. So -4 (p⃗×q⃗)=8î-40ĵ-24k̂, which gives p⃗×q⃗=-2î+10ĵ+6k̂.
4. Find lambda. a⃗×b⃗=(4-3λ)î-(3+2λ)ĵ-(λ²+2)k̂, so -2 a⃗×b⃗=(6λ-8)î+(4λ+6)ĵ+(2λ²+4)k̂. Matching this with -2î+10ĵ+6k̂ gives λ=1 in all three components.
5. So. |λ p⃗×q⃗|²=1²×(4+100+36)= 140, which is option (b).

The idea to keep. Whenever you see a⃗+b⃗ and a⃗-b⃗ together, remember that their cross product is -2 a⃗×b⃗ and their dot product is |a⃗|²-|b⃗|². Rename them first, then expand.

Answer from the NTA final answer key. Solution by PrepWiser.