Unduplicator

Last updated:

|Edit this page

This app prevents duplicate events from being ingested into PostHog. It's particularly helpful if you're backfilling information while you're already ingesting ongoing events.

How it works

The Unduplicator crafts an event UUID based on key properties for the event, so if the event is the same (see below for definition) it'll end with the same UUID.

When events are processed by ClickHouse, the database will automatically dedupe events which have the same toDate(timestamp), event, distinct_id and uuid combo, effectively making sure duplicates are not stored.

The app has two modes that define what's considered a duplicate event. Either mode will prevent duplicates within a project, though duplicates across projects are still permitted.

  • Event and Timestamp. An event will be treated as duplicate if the timestamp, event name and user's distinct ID matches exactly, regardless of what internal properties are included.
  • All Properties. An event will be treated as duplicate only if all properties match exactly, as well as the timestamp, event name and user's distinct ID.

Requirements

The Unduplicator requires either PostHog Cloud, or a self-hosted PostHog instance running version 1.30.0 or later.

Not running 1.30.0? Find out how to update your self-hosted PostHog deployment!

Installation

  1. Log in to your PostHog instance
  2. Click 'Apps' on the left-hand tool bar
  3. Search for 'Unduplicates'
  4. Select the Unduplicator app, press 'Install'.
  5. Once the app is installed, press the blue button to configure the app and select which of the de-duplication methods you want to use (described above).

Configuration

OptionDescription
Dedup Mode
Type: choice
Required: True

FAQ

Is the source code for this app available?

PostHog is open source and so are all apps on the platform. The source code for the Unduplicator app is available on GitHub.

Who created this app?

We'd like to thank former PostHog team member Paolo D'Amico for creating the Unduplicator. We miss you, Paolo!

Who maintains this app?

This app is maintained by the community. If you have issues with the app not functioning as intended, please let us know!

What if I have feedback on this app?

We love feature requests and feedback! Please tell us what you think! to tell us what you think.

What if my question isn't answered above?

We love answering questions. Ask us anything via our community forum, or drop us a message.

Questions?

Was this page useful?

Next article

Advanced GeoIP

This app extends functionality for the GeoIP Entricher app , offering the ability to automatically discard unwanted information in a way which is helpful users with privacy or compliance needs. Requirements This app requires either PostHog Cloud, or a self-hosted PostHog instance running version 1.30.0 or later. Not running 1.30.0? Find out how to update your self-hosted PostHog deployment ! You'll also need to have the GeoIP Entricher app installed. Installation Visit the 'Apps' page…

Read next article