Using element parameters as indices of sparse data

Sparse data exists, and sometimes needs a change of index. This article explains how element parameters can be used as LHS indices in assignments to make that conversion efficient — and covers what to do when the ideal case doesn't apply, plus a caveat to be aware of.
 

A small example

Consider a set S with index i, and a parameter P(i). The set S has 5 elements, and P is non-zero for 2 of them — so P is sparse.
 

There is also a set T with index j, and a parameter Q(j). For each j, the corresponding i is known and stored in element parameter ej2i(j).
 

Approach 1: element parameter on the RHS

! Convert P(i) to Q(j) via element parameter ej2i. Q(j) := P(ej2i(j));
 

This executes for every j:

• Determine ej2i(j) and store in a hidden temporary _ep_j
• Look up P(_ep_j)
• When non-zero: store the result

This requires O(card(T)) operations — one for each element of T.

Approach 2: element parameter on the LHS (inverse)

! Convert P(i) to R(j) via inverse element parameter ei2j. R(ei2j(i)) := P(i);
 

Here the running index is `i`, and the RHS is determined by the non-zero elements of `P`. Any zero in `P` maps to a zero in `R` — so zeros are never visited.

Execution:

• Visit each i where P(i) is non-zero
• Convert i to ei2j(i) before storing in R

This requires O(card(P)) operations — one per non-zero element in P.

When P is sparse, this is substantially faster than iterating over all of T.

---

A larger example

The small example may not feel compelling. Let's consider data at scale.

Data dimensions:

• 200,000 rows (sets s_A and s_B, each with 2×10⁵ elements)
• 2,000 columns (sets s_C and s_D, each with 2×10³ elements)
• 2 non-zero elements per row → 400,000 non-zeros total

The source data is bp_ac(i_a, i_c). The goal is to produce bp_bd(i_b, i_d) where indices are converted using element parameters.

Naive approach — element parameters on the RHS:
 

Running indices are [i_b, i_d]. For each combination, the engine must check whether the corresponding value in bp_ac exists. With 2×10⁵ × 2×10³ = 4×10⁸ combinations and only 4×10⁵ non-zeros, nearly all iterations are wasted.

Efficient approach — inverse element parameters on the LHS:

Running indices are [i_a, i_c] — the indices of the source data. The engine makes a single pass over the non-zero elements of bp_ac, converting each index before storing.

The above screen shots confirm theory.

---

Concern 1: what if inverse element parameters are not available?

If only ep_b2a(i_b) -> s_A is available (not the inverse ep_a2b), it is straightforward to construct a binary parameter:

bp_ab(ep_b2a(i_b), i_b) := 1;

Similarly for bp_cd(i_c, i_d). These binary parameters encode the same mapping information.

First attempt — naive reformulation using binary parameters:

The index order here is [i_b, i_d, i_a, i_c]. Starting from [i_b, i_d], no identifier constrains the valid values of i_d for a given i_b, so every i_d is tried for every i_b. Even though each probe is cheap, the total is O(card(s_B) × card(s_D)) operations.

Better approach — reorder the indices:

With index order [i_a, i_b, i_c, i_d], the execution engine can exploit the structure:

• - Given `i_a`, `bp_ab(i_a, i_b)` determines the relevant values of `i_b`
• - Given `[i_a, i_b]`, `bp_ac(i_a, i_c)` determines the relevant values of `i_c`
• - Given `[i_a, i_b, i_c]`, `bp_cd(i_c, i_d)` determines the relevant values of `i_d`

Each step is guided — no combinations are tried blindly. This runs sub-second.

Next, aggregate into the final result:

The execution engine creates a permutation [i_b, i_d, i_a, i_c] using O(card(_bp_bd4h)) operations, making this summation efficient and sub-second.

The combined execution of both assignments remains sub-second.

> **Tip:** Declare `_bp_bd4h` as a local parameter in the procedure. It is then automatically emptied when the procedure finishes — no manual cleanup needed.

---

Concern 2: what if bp_bd2 already has data?

This one can come as an unwelcome surprise.

If bp_bd2 already contains data and you execute the assignment a second time:
 

When a parameter already has data, the execution engine must ensure that any zero computed on the RHS results in a deletion on the LHS. Because the LHS uses element parameters (not plain indices), this is not a straightforward pass over existing data — additional work is required to locate and remove stale values.

The remedy is simple: empty bp_bd2 before the assignment:

Adding an empty statement resets the parameter first, so the subsequent assignment runs as efficiently as the first time.

Summary

When inverse element parameters are available, use them on the LHS — it is the most concise and efficient solution. When they are not, construct binary parameters and pay careful attention to index order.

Either way, remember to empty the target parameter before re-running the assignment if it may already contain data.