How to Import CSV Into Elasticsearch

An Elasticsearch index takes a document that declares nothing and works out the rest. "The automatic detection and addition of new fields is called dynamic mapping." A string becomes text with a .keyword sub-field. A whole number becomes long. A string that matches a date pattern becomes a date, because date detection is on by default.

Then the guess stands. Elastic states the consequence in one sentence. "Except for supported mapping parameters, you can't change the mapping or field type of an existing field." The way back is a new index and a reindex of everything in the old one.

So the first document through the door writes the schema for every document after it. In an import feature, the first document is built from a file somebody else sent you.

The file that arrives

A seed bank runs a public search over its collection. The partner banks that collect for it send accession exports, each out of whatever field system that partner keeps.

1Accession No,Scientific Name,Vernacular,Date Collected,Cntry,Locality,Storage,Viability %,Seed Count,Donor2NV-2026-0117,Silene stenophylla,narrow-leaved campion,03/04/2026,ru,"Kolyma river terrace, 62 m",orthodox,92,"1,450",Kolyma Field Station3NV-2026-0118,Papaver radicatum,arctic poppy,29/09/2025,NO,"Adventdalen, scree slope",Orth.,88.5,860,Nordvik Survey4NV-2026-0119,Quercus robur,pedunculate oak,17/11/2025,GB,"Wistman's Wood, granite clitter",Recalcitrant,n/a,240,Devon Partner Bank5NV-2026-0117,Silene stenophylla,narrow-leaved campion,03/04/2026,RU,"Kolyma river terrace, 62 m",Intermed.,92,"1,450",Kolyma Field Station⋮91 rows not shown97NV-2026-0212,Dryas octopetala,mountain avens,22/03/2026,IS,"Skaftafell, lava field",orthodox,95,"2,100",Nordvik Survey

The header row says Cntry where the collection says country, and Vernacular where it says common name. Storage behaviour arrives as orthodox, Orth., Recalcitrant and Intermed.. Dates run day first. NV-2026-0117 appears twice, once orthodox and once intermediate, because the partner revised the storage class and left both lines in.

Donor is the column the collection has no field for.

The person drags the file into the importer inside your app. Updog Importer reads it in the browser, matches the headers to your schema, holds the storage classes to your list, and puts every row in front of them. Your onComplete handler posts the clean rows to a route you own. That route holds the API key and calls the bulk API.

No Updog server stands between the browser and the cluster.

The index and what it declares

Every field the collection cares about is declared before a single document arrives.

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "accession":   { "type": "keyword" },
      "taxon": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }
      },
      "commonName":  { "type": "text" },
      "collectedOn": { "type": "date", "format": "strict_date_optional_time" },
      "country":     { "type": "keyword" },
      "locality":    { "type": "text" },
      "storage":     { "type": "keyword" },
      "viability":   { "type": "float" },
      "seedCount":   { "type": "integer" },
      "extra":       { "type": "flattened" }
    }
  }
}

"dynamic": "strict" is the line that keeps the mapping yours. Elastic describes what it does plainly. "If new fields are detected, an exception is thrown and the document is rejected." A partner's spare column can no longer mint a field.

taxon is text with a keyword sub-field, because the collection searches on the species name and also sorts by it. Keyword fields are "often used in sorting, aggregations, and term-level queries, such as term", and Elastic's own advice for the other job is "Avoid using keyword fields for full-text search. Use the text field type instead." ignore_above at 256 keeps a runaway string out of the sorted copy while leaving it in _source.

collectedOn names one format. A bare date field defaults to "strict_date_optional_time||epoch_millis", and the strict_ family is exact about digits, where "year, month and day parts of the month must use respectively 4, 2 and 2 digits exactly, potentially prepending zeros". Updog stores every parsed date as YYYY-MM-DD, so the two line up.

extra is flattened, and the section on the created column is where it earns its place.

What happens to a field nobody declared

Under the default setting, a new field gets a type from the value that carried it. Elastic publishes the whole table. true and false become boolean. A decimal becomes float. A whole number becomes long. A string with no date or numeric match becomes "text with a .keyword sub-field". Numeric detection is off by default, so "92" stays a string.

Two costs follow. The first is the mapping you cannot change, which sends you to a new index and a reindex. The second has a number on it. index.mapping.total_fields.limit defaults to 1000, and "Field and object mappings, field aliases, and mapped runtime fields all count towards this limit." Elastic has a name for what happens next. "Runaway field growth is colloquially called 'mapping explosion'."

The dynamic setting has four values, and each one is a different answer to the same partner column. true adds the field. runtime adds it as a runtime field, "not indexed and are loaded from _source at query time". false ignores it, so it "will not be indexed or searchable". strict rejects the document.

This collection takes strict and gives the spare columns one field of their own. Under flattened, "the entire object is mapped as a single field", which "can help prevent a mappings explosion from having too many distinct field mappings". Leaf values inside it are "indexed as string keywords, without analysis or special handling for numbers or dates". A donor name stays findable by its exact value and the mapping stays the size you wrote it.

The schema in Updog Importer

The columns array carries the nine fields the index declares beside extra.

import type { DataEditorColumn } from "@updog/data-editor";

const STORAGE = ["Orthodox", "Recalcitrant", "Intermediate"];

export const columns: DataEditorColumn[] = [
  {
    id: "accession",
    title: "Accession",
    size: 150,
    transformer: (value) => String(value).trim().toUpperCase(),
    validators: [
      { type: "required" },
      { type: "regex", pattern: "^NV-\\d{4}-\\d{4}$" },
      { type: "unique" },
    ],
  },
  {
    id: "taxon",
    title: "Scientific name",
    size: 220,
    validators: [
      { type: "required" },
      { type: "function", fn: (value) => String(value).length > 256
        ? { level: "error", message: "256 characters at most" }
        : null },
    ],
  },
  { id: "commonName", title: "Common name", size: 200 },
  {
    id: "collectedOn",
    title: "Collected on",
    size: 150,
    editor: { type: "date" },
    validators: [
      { type: "required" },
      { type: "date" },
    ],
  },
  {
    id: "country",
    title: "Country",
    size: 110,
    transformer: (value) => String(value).trim().toUpperCase(),
    validators: [
      { type: "required" },
      { type: "regex", pattern: "^[A-Z]{2}$" },
    ],
  },
  { id: "locality", title: "Locality", size: 260 },
  {
    id: "storage",
    title: "Storage behaviour",
    size: 170,
    editor: { type: "select", options: STORAGE, enableCustomValue: false },
    validators: [{ type: "required" }, { type: "oneOf", values: STORAGE }],
  },
  {
    id: "viability",
    title: "Viability",
    size: 130,
    editor: { type: "number" },
    validators: [
      { type: "number", min: 0, max: 100, decimalPlaces: 1 },
    ],
  },
  {
    id: "seedCount",
    title: "Seed count",
    size: 140,
    editor: { type: "number" },
    validators: [
      { type: "required" },
      { type: "number", min: 1 },
    ],
  },
];

Each rule answers a line in the mapping. The regex on the accession holds it to the collection's own format, which also keeps it far inside the 512 bytes an _id allows. The function validator counts the species name against 256 characters, the same figure ignore_above uses, so a string that would fall out of the sorted copy is reported in a grid cell. number holds viability between 0 and 100.

{ type: "unique" } on the accession is the rule that saves the write. Uniqueness is checked against every other row in the same column, so both NV-2026-0117 cells light up before submit. That matters because Elasticsearch answers a duplicate id with silence. The index action "adds or replaces a document as necessary", so the second row wins and the bulk response reports two successes.

The select on storage behaviour is closed, and n/a in the viability column is caught by { type: "number" }. A value like that reaching a float field is refused by the index, since ignore_malformed defaults to false. Turn it on and the field is dropped instead, its name recorded in the _ignored field and its value left out of the index, where no search reaches it.

The column your schema never had

enableCreateColumn is on by default, and it is what a person does with Donor.

An unmatched header opens a select in the column matching step, listing your fields and one more entry underneath a divider. Create column. The person names it, and the file column keeps its data. Once the grid opens, that column has a context menu of its own, so it can be renamed or dropped.

The created column then arrives on the result as a key your schema never declared. Your handler compares each key against the ids you wrote and puts the rest in one place. Everything the partner brought stays searchable, and the index gains no field, because flattened maps the whole object as one.

The alternative is the same decision made by a machine that cannot take it back. A donor name reaching a dynamic index becomes text with a .keyword sub-field for the life of that index.

The headers the partner sends

Updog scores each header against the column id and the column title, and the higher score wins.

Cntry is the interesting one. Neither string sits inside the other, so the contains tier refuses, no whole word is shared, and the built-in country group carries nation and countryname without cntry. Two insertions sit inside the two edits allowed at that length, so the abbreviation lands anyway. Vernacular shares nothing with commonName at any tier, so it needs the one synonyms entry in the mount below.

Values follow the same ladder. orthodox and Recalcitrant are exact. Orth. normalizes to four characters, which is the floor of the contains tier, and Orthodox contains it. Intermed. reaches Intermediate the same way.

The file settles its own dates. 29/09/2025 and 17/11/2025 both carry a value above twelve in the first position, so the scan reads the whole file day first, and 03/04/2026 follows as the third of April.

The mount

The props tie the file, the schema and the index together.

<DataEditor<SeedAccession>
  apiKey="your-license-key"
  open={open}
  onClose={closeEditor}
  columns={columns}
  primaryKey="accession"
  enableCreateColumn
  enableDeleteRow="all"
  blockSubmitOnError
  synonyms={{
    columns: {
      commonName: ["vernacular", "vernacular name", "local name"],
    },
    values: {
      Orthodox: ["long term", "desiccation tolerant"],
      Recalcitrant: ["recalc", "desiccation sensitive"],
    },
  }}
  onComplete={onComplete}
/>

primaryKey is the accession alone, and it becomes the _id of the document. There is no upsert to configure here, since one index action creates a document that is new and replaces a document that is there.

enableCreateColumn is written out even though it is the default, because the whole donor column depends on it. enableDeleteRow="all" lets the person drop the stale NV-2026-0117 line once the grid shows both. blockSubmitOnError keeps submit disabled while any row carries an error. Whatever they fix by hand comes back as learnedSynonyms, two lists of { source, target } pairs to store and fold into the synonyms tables next season.

Every snippet here is React. Those props reach Vue, Angular and Svelte through the web component build. The install and the modal wiring beneath this mount live in how to import a CSV file into a React app.

The result on submit

On submit, Updog Importer hands your handler every row grouped by source, each one carrying four independent flags.

import type { DataEditorResult, ResultRow } from "@updog/data-editor";

const CHUNK_SIZE = 2000;
const DECLARED = new Set(columns.map((column) => column.id));

const readLabel = (key: string) => {
  return key
    .replace(/^__dynamic__:/, "")
    .replace(/-[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}$/, "");
};

const extraFields = (row: SeedAccession) => {
  const extra: Record<string, unknown> = {};
  for (const [key, value] of Object.entries(row)) {
    if (DECLARED.has(key) || value === "" || value == null) continue;
    extra[readLabel(key)] = value;
  }
  return extra;
};

const toDocument = (row: SeedAccession) => ({
  accession: row.accession,
  taxon: row.taxon,
  commonName: row.commonName,
  collectedOn: row.collectedOn,
  country: row.country,
  locality: row.locality,
  storage: row.storage,
  viability: row.viability === "" ? null : Number(row.viability),
  seedCount: Number(row.seedCount),
  extra: extraFields(row),
});

const toAction = (entry: ResultRow<SeedAccession>) => {
  if (entry.isDeleted && entry.isNew) return [];
  if (entry.isDeleted) {
    return [{ op: "delete", accession: entry.row.accession }];
  }
  return [{
    op: "index",
    accession: entry.row.accession,
    document: toDocument(entry.row),
  }];
};

const onComplete = useCallback(async (result: DataEditorResult<SeedAccession>) => {
  for (const source of result.sources) {
    const actions = source.rows.flatMap(toAction);

    for (let start = 0; start < actions.length; start += CHUNK_SIZE) {
      const answer = await fetch("/api/accessions/index", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          actions: actions.slice(start, start + CHUNK_SIZE),
          last: start + CHUNK_SIZE >= actions.length,
        }),
      });
      if (!answer.ok) throw new Error((await answer.json()).message);
    }
  }
  await storeSynonyms(result.learnedSynonyms);
}, []);

A new accession and an edited accession become the same action, since one index call decides which of the two it is. A row the person added and then deleted goes nowhere. Anything the person created shows up in extra under the name they typed, with the generated suffix trimmed off the key.

Two boundaries size this write, and they belong to different parties. The chunk of two thousand is yours, sized against what your own route accepts. The size of the Elasticsearch request belongs to the client, which flushes its body at five million bytes by default. Elastic publishes no row count at all. "There is no 'correct' number of actions to perform in a single bulk request." What it does publish is a ceiling of 100mb on an HTTP request, and one line of advice, that "it is advisable to avoid going beyond a couple tens of megabytes per request".

Throw when a route answers with a failure. Updog holds submit until your promise resolves, then empties the editor. A handler that traps the error and returns counts as a finished import, and the whole export clears the grid unindexed. A thrown error keeps the grid as it stands, with every mapping and hand correction on it. The person submits again on rows that never left the screen.

The route that indexes the rows

The route holds the API key, and the browser stops there.

import { Client, errors } from "@elastic/elasticsearch";

const client = new Client({
  node: process.env.ELASTIC_NODE,
  auth: { apiKey: process.env.ELASTIC_API_KEY },
});

const INDEX = "seed-accession";

const droppedId = (operation: unknown) => {
  const [line] = Object.values(operation as Record<string, { _id: string }>);
  return line._id;
};

app.post("/api/accessions/index", async (request, response) => {
  const session = await getVerifiedSession(request);
  if (!session) return response.status(401).json({ message: "Not signed in" });
  if (!session.canEditCollection) {
    return response.status(403).json({ message: "No collection access" });
  }

  const lost: { accession: string; status: number; reason: string }[] = [];

  try {
    const stats = await client.helpers.bulk({
      datasource: request.body.actions,
      onDocument: (action) => {
        if (action.op === "delete") {
          return { delete: { _index: INDEX, _id: action.accession } };
        }
        return [
          { index: { _index: INDEX, _id: action.accession } },
          action.document,
        ];
      },
      onDrop: (dropped) => {
        lost.push({
          accession: droppedId(dropped.operation),
          status: dropped.status,
          reason: dropped.error?.reason ?? "no document with that id",
        });
      },
      refreshOnCompletion: request.body.last ? INDEX : false,
    });

    if (lost.length > 0) {
      request.log.error({ lost });
      return response.status(502).json({
        message: "The index refused " + lost.length + " of these rows",
        lost,
      });
    }

    response.json({ indexed: stats.successful });
  } catch (error) {
    if (error instanceof errors.ResponseError) {
      request.log.error({ status: error.statusCode, type: error.message });
    } else {
      request.log.error({ type: String(error) });
    }
    return response.status(502).json({ message: "Elasticsearch refused the batch" });
  }
});

getVerifiedSession() stands in for your own server-side authentication check, and the collection permission is read before a document is built. An Elasticsearch API key is the credential for this seat, one of the "token-based authentication mechanisms designed to authenticate applications and services accessing Elasticsearch", and it can be narrowed, since "you can opt to configure access to specific Elasticsearch APIs and resources by assigning the key with predefined roles or custom privileges".

The bulk API speaks newline-delimited JSON, where an action line is followed by a document line, a delete stands as one line on its own, and "The final line of data must end with a newline character". The helper builds that body from what onDocument returns, so the route writes actions and the client writes NDJSON. Deletes ride in the same request as the writes.

The 200 that carries failures

A bulk request that fails halfway still answers with HTTP 200. The verdict sits in the body, on a flag Elastic describes as true when "one or more of the operations in the bulk request did not complete successfully", with a status and an error on each item that failed.

The helper reads that array for you and splits it two ways. A document rejected with 429 goes back into the queue, and the client's own source says why. "429 is the only status code where we might want to retry a document, because it was not an error in the document itself, but the ES node was handling too many operations." That is the code Elasticsearch answers with under load, "TOO_MANY_REQUESTS (429) response codes", and the advice beside it is to "pause indexing a bit before trying again, ideally with randomized exponential backoff".

Everything else reaches onDrop, which receives the status, the error, the action line, whether it had already been retried, and, on a write, the document that followed the action. A delete has no document line to hand back, so the accession has to come off the action, where the id sits either way. One case arrives with no error either. A delete whose document was never there answers "result": "not_found" with "status": 404 and nothing else, so the reason has to be written by you.

A request the cluster refuses outright throws instead. The client raises a ResponseError whose statusCode holds the HTTP status and whose message holds the error type Elasticsearch named. Log both, answer the browser with a sentence a curator can act on, and let the handler throw.

The rows nobody can find yet

An indexed document is durable before it is findable. Search runs over segments, and "A refresh makes all operations performed on an index since the last refresh available for search."

Elasticsearch does this on its own, with one condition worth knowing. "By default, Elasticsearch periodically refreshes indices every second, but only on indices that have received one search request or more in the last 30 seconds." A collection index that nobody has searched this minute sits unrefreshed, and the curator who just imported four thousand accessions searches for one and finds nothing.

So the last chunk asks for a refresh by name. refreshOnCompletion takes the index string and runs one refresh after the final flush, which is why the browser marks the last request. The same result is available per request through the refresh parameter, where wait_for will "Wait for the changes made by the request to be made visible by a refresh before replying". One refresh at the end of an import costs less than one on every chunk.

Importing a CSV into ClickHouse carries the other version of this gap, where the write is visible immediately and the deduplication behind it happens later.

The parts nobody ships for you

Updog Importer integrates with nobody. There is no Elasticsearch connector, no destination list, no webhook and no server of ours. onComplete hands your code an object, and the index, the route and the API key in the middle are work you do.

Elasticsearch already ships its own way in for the other case. Kibana's file upload takes a CSV, works out the fields, and lands them in an index, and Elastic states the scope itself. "The upload feature is not intended for use as part of a repeated production process, but rather for the initial exploration of your data." Logstash covers the repeated case for files a server can reach. For a file your own team assembled, both are the shorter way in. Everything above exists for the export a partner sends, opened in a browser inside a session your app issued. Client-side and server-side CSV import names the jobs each of the two models fits.

The pieces you wrote and the next partner

You wrote one mapping that refuses a field it never declared, a schema with nine columns, one synonyms block, a handler that sorts declared keys from created ones, and a route that indexes a chunk and refreshes once at the end. The file stays on the machine that opened it. The rows travel from your own front end to your own route and into Elasticsearch. The only party you added to the chain is yourself. Point the same setup at a React CSV importer modal or at the web component and the middle stays the same.

The next partner sends a spreadsheet with two spare columns and a fourth spelling of Orthodox. The mappings from this season are already stored, the spelling costs one more line in synonyms, and the spare columns land in one field that was there before they arrived.