JSON Deduplicate – Remove Duplicates from JSON Arrays
Remove duplicate array values, including nested arrays, while preserving the first occurrence.
Remove duplicate array values, including nested arrays, while preserving the first occurrence.
JSON Deduplicate removes repeated items from arrays while keeping the first occurrence. It walks the whole document, so duplicates inside nested arrays are cleaned too, and the objects and keys around them are left as they were. The output is pretty-printed and stays on your device.
Typical inputs are tag lists, ID lists, merged API pages that overlap, imported records and test fixtures that were assembled by copy and paste.
Arrays are processed from the inside out: nested values are deduplicated first, then each item is turned into its JSON text and compared with the items already seen. Two items are duplicates only when that text is identical. In practice that means:
1 and 1.0 are the same (both parse to the number 1), but 1 and "1" are not."Tools", "tools" and "tools " are three values.{"tags":["json","api","json","tools","api"],"ids":[3,1,3,2,1]}Output:
{
"tags": [
"json",
"api",
"tools"
],
"ids": [
3,
1,
2
]
}[{"id":1,"email":"ada"},{"id":2,"email":"bo"},{"id":1,"email":"ada"},{"email":"bo","id":2},{"id":"2","email":"bo"}]Output:
[
{
"id": 1,
"email": "ada"
},
{
"id": 2,
"email": "bo"
},
{
"email": "bo",
"id": 2
},
{
"id": "2",
"email": "bo"
}
]Only the exact repeat of record 1 is removed. The reordered copy of record 2 and the version whose id is a string both survive, because their JSON text differs.
{"route":[{"point":[0,0]},{"point":[5,0]},{"point":[0,0]}],"labels":[["a","a"],["a"]]}Output:
{
"route": [
{
"point": [
0
]
},
{
"point": [
5,
0
]
}
],
"labels": [
[
"a"
]
]
}The repeated route point is removed as intended, but the coordinate pair [0,0] also collapses to [0] because it is an array with a repeated value. Do not run whole documents containing tuples through the tool without checking them.
This page compares whole values; it has no option to keep one record per id or email. When that is what you need, a two-line JavaScript filter keeps the first record for each key:
const seen = new Set();
const unique = rows.filter(r => !seen.has(r.id) && seen.add(r.id));The jq equivalent is unique_by(.id), which also sorts the result by that key. In Python, iterate and track seen keys in a set in the same way.
offset and limit, the last items of one page reappear at the top of the next. Exact repeats like these are what this tool removes cleanly.To see how many items were dropped, compare array lengths before and after: the JSON viewer shows the item count of each array.
Deduplication often follows a combine step: concatenate two exports, then clean them here. When the inputs are objects rather than arrays, use JSON Merge instead. Turn newline-delimited logs into an array with JSONL to JSON before removing repeated events, and confirm what was removed with the JSON diff checker. When the cleaned list is headed for a spreadsheet, finish with JSON to CSV.
Paste the JSON, click Deduplicate and copy the output. Every array in the document keeps only the first copy of each value, and the order of the remaining items does not change.
Only if the whole objects are identical. Comparison uses the complete value, so records that share an id but differ in any other field are both kept. Use a small script or jq unique_by(.id) to deduplicate by key.
Objects are compared by their serialised JSON, which includes key order. {"id":1,"n":"a"} and {"n":"a","id":1} count as different. Types matter too: 2 and "2" are different values.
It processes every array at every depth. Coordinate pairs such as [0, 0] or other tuples with repeated values will be shortened, so check the output if your data contains them.
No. The first occurrence of each value stays where it was and later repeats are removed, so the original order is preserved.
No. Deduplication runs in your browser and the data never leaves the page.