NEPSE Data Pipeline: Ankit Sanjyal

NEPSE Data Pipeline

Live and historical market data from the Nepal Stock Exchange, landed in DuckDB and Parquet with validation at every layer boundary. Every constraint below was measured against the live API, not taken from documentation, because for one of these datasets, a day you miss is gone permanently.

Data Engineering DuckDB Parquet Pydantic GitHub Actions Hugging Face Python

GitHub Dataset Notebooks

42,709

contracts / day

516,620

deep-history rows

1995–2026

daily series range

100.00%

volume reconciled

The Constraint That Shaped Everything

NEPSE's floorsheet, the record of every executed contract, is the only dataset here with no second chance. The API serves the current session and nothing else: it ignores the businessDate parameter entirely, and the per-company route is nginx-403'd. A day the poller misses is not recoverable from NEPSE, from a vendor, or from anywhere.

It is also the most valuable table in the warehouse. A normal session is 42,709 contracts, each carrying a microsecond tradeTime and both the buyer and seller broker IDs. Most exchanges never expose broker identity at all.

Why that changes the architecture. Because the data cannot be re-fetched, raw responses are written to bronze before anything is parsed, if the parser is wrong, the bytes are still on disk and silver can be rebuilt. The same reasoning is why the capture agent runs with KeepAlive, and why a failed run is never allowed to look like an empty one.

Layers

bronze/   raw gzipped JSON, one file per response, immutable
          -> data/bronze/{dataset}/date=YYYY-MM-DD/{epoch_ms}-p{page}.json.gz
silver/   validated, typed, deduplicated Parquet, one file per table per day
gold/     bars_1m, broker_flow_daily, daily_stats

DuckDB is the working warehouse; Parquet is the durable export. At roughly 12 GB a year the whole thing fits on one machine, so Spark or Kafka here would be cost with no benefit. Making Parquet the source of truth is also what lets the pipeline run on a CI runner that is destroyed minutes later, a warehouse rebuilt from Parquet alone reproduced all five tables exactly, floorsheet turnover included, to the paisa.

What the API Actually Does

Every row below was measured against the live endpoint. Several are load-bearing design constraints rather than trivia, the ones about pagination and the trading week are silent data-loss bugs if you assume the documented behaviour.

Measured fact	Consequence
No streaming feed exists; the site polls REST, and the backend refreshes about every 24s.	“Live” means polling. Faster than ~20s buys nothing.
A browser `User-Agent` is mandatory. Without one the connection completes TLS and then hangs forever.	No 403 and no error, every client must set it or die silently.
Auth is a WASM-derived token: the server returns 5 salts, NEPSE's own `css.wasm` turns them into character offsets, TTL is 45 seconds.	Isolated behind one token source so the rest of the code never sees it.
`sort=contractId,desc` is not honoured across pages. Page 10 was observed starting at one id and ending above it.	A watermark “stop at a known id” would silently skip trades. Not used.
Pages intermittently return a bare `[]`, then yield 500 rows on retry.	Treated as transient, never as end-of-data.
All history endpoints clamp to a rolling 365 days, server-side, even given an explicit older start date.	A hard floor on backfill, and the reason for a second source.
The intraday index series is wiped a few hours after close: 610 points at 19:30 NPT, zero by 20:25.	Must be captured during or right after the session.
NEPSE traded Sunday–Thursday until about 2026-04-06, then Monday–Friday.	A `weekday() < 5` weekend filter silently drops ~33 real trading days.

Validation

Bad data fails loudly. It is never dropped and never let through.

Pydantic contracts with extra="forbid", a new upstream field raises, so schema drift surfaces immediately instead of as a column of nulls months later.
Row-level tripwires, qty × rate == amount within one rupee, positive prices, trade_time matching business_date, high >= low.
Drift detection, more than 10% of rows failing on a page raises rather than writing the survivors.
Quarantine, rejected rows are written out with the reason attached, not discarded.
Flag, don't drop, some debentures legitimately report a close outside the day's traded range. Those rows are kept and flagged: dropping them would lose real securities, accepting them silently would hide a real anomaly.
Completeness gate, a session is only published once the row count matches upstream's totalElements.
Reconciliation, gold must conserve floorsheet volume and turnover exactly, or the build raises.

Deep History, and How Far to Trust It

NEPSE's 365-day floor is hard, so a second source (ShareSansar) supplies the daily series much further back. It is landed in a separate table, tagged with its origin, and never merged into the NEPSE tables, 516,620 rows across 653 symbols, from 1995-07-20 to 2026-08-19, in a single 10.3 MB Parquet file.

The wider universe is the point. A symbol list drawn from the rolling window alone is survivorship-biased: 175 symbols exist only in the second source, delisted mutual funds, converted promoter shares, long-suspended names, and would otherwise be invisible.

The 225-day overlap is the only place a second source can be checked against an authoritative one:

Check	Result
Overlapping rows	75,671, every NEPSE row has a match
Volume exact	75,671 / 75,671, 100.00%
Close exact	75,657 / 75,671, 99.98%

Fourteen closes disagree: four are debentures already carrying the out-of-range flag, and the rest differ by under 1% on a single session, which looks like a last-trade-versus-close convention rather than corruption. Volume agreeing on every single row is the stronger signal, the two sources are reporting the same trades. That earns confidence in the mechanism, not a guarantee about 1998.

Three Worked Analyses

The notebooks are committed with their outputs, so the findings can be read without running anything.

Notebook	Finding
Who is actually trading this stock?	Large broker positions repeat direction 71.8% of the time next session, with no next-day price edge detectable yet.
The corporate actions trap	NEPSE's ~10% circuit limit doubles as a corporate-action detector; 747 candidates recovered from prices alone.
The symbols that disappeared	36% of NEPSE symbols are delisted; survivorship bias measured at +0.74 pts/year.

Hosted Capture

Capture runs on GitHub Actions and publishes to Hugging Face, both free at this scale. The job runs after close rather than as a live poller: NEPSE keeps serving the current session's floorsheet for hours afterwards, measured complete at +2h and again at +5.5h, wiped by +5.9h, so one short run captures the same contracts a four-hour poller would at a twentieth of the compute. The intraday index sets the deadline, since it disappears about 4h after close.

The workflow fires twice because scheduled GitHub jobs are routinely delayed and occasionally dropped. Every write is an upsert, so the second run is a no-op when the first succeeded.

import duckdb
duckdb.sql("SELECT * FROM 'hf://datasets/AnkitSanjyal/nepse-market-data/floorsheet/**/*.parquet'")

Known Gaps

Stated plainly, because a dataset is only usable if you know where it stops being trustworthy.

Corporate actions are not adjusted. Bonus shares and rights issues are common on NEPSE and leave fake gaps in any price series. An adjustment table is required before this data is fit for modelling.
History beyond 365 days is second-source, tagged as such, and trustworthy only as far as the reconciliation above justifies.
Intraday index history before today cannot be recovered, the same capture-or-lose-it property as floorsheet.
Market depth and the order book are not collected. One request per symbol, deliberately deferred.
The WASM token will break when NEPSE rotates the module. The failure is loud (401s); a browser-driven fallback is the intended remedy and is not built yet.