JSON Deduplicate – Remove Duplicates from JSON Arrays

Remove duplicate array values, including nested arrays, while preserving the first occurrence.

Remove duplicate values from JSON arrays

JSON Deduplicate removes repeated items from arrays while keeping the first occurrence. It walks the whole document, so duplicates inside nested arrays are cleaned too, and the objects and keys around them are left as they were. The output is pretty-printed and stays on your device.

Typical inputs are tag lists, ID lists, merged API pages that overlap, imported records and test fixtures that were assembled by copy and paste.

How to use it

  1. Paste a JSON array, or any document that contains arrays, into Input JSON.
  2. Click Deduplicate.
  3. Review the Output. Repeats are gone, the first copy of each value remains in its original position, and invalid JSON shows an error instead.
  4. Click Copy to use the cleaned result.

How duplicates are detected

Arrays are processed from the inside out: nested values are deduplicated first, then each item is turned into its JSON text and compared with the items already seen. Two items are duplicates only when that text is identical. In practice that means:

Worked examples

1. Tag and ID lists

{"tags":["json","api","json","tools","api"],"ids":[3,1,3,2,1]}

Output:

{
  "tags": [
    "json",
    "api",
    "tools"
  ],
  "ids": [
    3,
    1,
    2
  ]
}

2. An array of records

[{"id":1,"email":"ada"},{"id":2,"email":"bo"},{"id":1,"email":"ada"},{"email":"bo","id":2},{"id":"2","email":"bo"}]

Output:

[
  {
    "id": 1,
    "email": "ada"
  },
  {
    "id": 2,
    "email": "bo"
  },
  {
    "email": "bo",
    "id": 2
  },
  {
    "id": "2",
    "email": "bo"
  }
]

Only the exact repeat of record 1 is removed. The reordered copy of record 2 and the version whose id is a string both survive, because their JSON text differs.

3. Nested arrays, including a pitfall

{"route":[{"point":[0,0]},{"point":[5,0]},{"point":[0,0]}],"labels":[["a","a"],["a"]]}

Output:

{
  "route": [
    {
      "point": [
        0
      ]
    },
    {
      "point": [
        5,
        0
      ]
    }
  ],
  "labels": [
    [
      "a"
    ]
  ]
}

The repeated route point is removed as intended, but the coordinate pair [0,0] also collapses to [0] because it is an array with a repeated value. Do not run whole documents containing tuples through the tool without checking them.

Removing duplicates by a key

This page compares whole values; it has no option to keep one record per id or email. When that is what you need, a two-line JavaScript filter keeps the first record for each key:

const seen = new Set();
const unique = rows.filter(r => !seen.has(r.id) && seen.add(r.id));

The jq equivalent is unique_by(.id), which also sorts the result by that key. In Python, iterate and track seen keys in a set in the same way.

Where duplicate JSON items come from

To see how many items were dropped, compare array lengths before and after: the JSON viewer shows the item count of each array.

Related tools

Deduplication often follows a combine step: concatenate two exports, then clean them here. When the inputs are objects rather than arrays, use JSON Merge instead. Turn newline-delimited logs into an array with JSONL to JSON before removing repeated events, and confirm what was removed with the JSON diff checker. When the cleaned list is headed for a spreadsheet, finish with JSON to CSV.

FAQ

How do I remove duplicates from a JSON array?

Paste the JSON, click Deduplicate and copy the output. Every array in the document keeps only the first copy of each value, and the order of the remaining items does not change.

Does it treat two objects with the same id as duplicates?

Only if the whole objects are identical. Comparison uses the complete value, so records that share an id but differ in any other field are both kept. Use a small script or jq unique_by(.id) to deduplicate by key.

Why were two identical-looking objects both kept?

Objects are compared by their serialised JSON, which includes key order. {"id":1,"n":"a"} and {"n":"a","id":1} count as different. Types matter too: 2 and "2" are different values.

Will it change arrays I did not mean to touch?

It processes every array at every depth. Coordinate pairs such as [0, 0] or other tuples with repeated values will be shortened, so check the output if your data contains them.

Does it sort the result?

No. The first occurrence of each value stays where it was and later repeats are removed, so the original order is preserved.

Is my JSON uploaded?

No. Deduplication runs in your browser and the data never leaves the page.