Just for tolerance of the crystal there are 20ppm. For aging additional 3ppm/year. This are vaues at exact 25 degree. If the crystal temperature is rising just to 45 degree on you working system you must add 16ppm.
So it is impossible to reduce offset without temperature compensation.
This would need a always working board to measure the temperature near the crystal - impossible.
So the best way for more exact values to use a temperature controlled oscillator (TXCO) on I2C. We have a good experience with Maxim DS3231 with just 3.5ppm between -40...85 degree. This means daily offset is max. 0.3 sec.