PyCall.jl icon indicating copy to clipboard operation
PyCall.jl copied to clipboard

Converting a numpy.datetime64 into DateTime?

Open Datseris opened this issue 5 years ago • 2 comments

I load some data using Python's xarray, which result to datetime-vectors, for example:


julia> x = w["time"].values
PyObject array(['2000-01-01T00:00:00.000000000', '2000-02-01T00:00:00.000000000',
       '2000-03-01T00:00:00.000000000', '2000-04-01T00:00:00.000000000',
       '2000-05-01T00:00:00.000000000', '2000-06-01T00:00:00.000000000',
       '2000-07-01T00:00:00.000000000', '2000-08-01T00:00:00.000000000',
       '2000-09-01T00:00:00.000000000', '2000-10-01T00:00:00.000000000',
       '2000-11-01T00:00:00.000000000', '2000-12-01T00:00:00.000000000',
       '2001-01-01T00:00:00.000000000', '2001-02-01T00:00:00.000000000',
       '2001-03-01T00:00:00.000000000', '2001-04-01T00:00:00.000000000',
       .....
       '2019-07-01T00:00:00.000000000', '2019-08-01T00:00:00.000000000',
       '2019-09-01T00:00:00.000000000', '2019-10-01T00:00:00.000000000',
       '2019-11-01T00:00:00.000000000', '2019-12-01T00:00:00.000000000',
       '2020-01-01T00:00:00.000000000', '2020-02-01T00:00:00.000000000',
       '2020-03-01T00:00:00.000000000', '2020-04-01T00:00:00.000000000',
       '2020-05-01T00:00:00.000000000', '2020-06-01T00:00:00.000000000',
       '2020-07-01T00:00:00.000000000', '2020-08-01T00:00:00.000000000',
       '2020-09-01T00:00:00.000000000'], dtype='datetime64[ns]')

julia> y = x[1]
PyObject numpy.datetime64('2000-01-01T00:00:00.000000000')

The data are not automatically put into Julia form, which happens if the type of w["dimension"] is Float64 or Integer. But that's okay, I don't mind converting myself. The problem is, that is not possible either:

julia> DateTime(y)
ERROR: MethodError: no method matching Int64(::PyObject)
Closest candidates are:
  Int64(::Union{Bool, Int32, Int64, UInt32, UInt64, UInt8, Int128, Int16, Int8, UInt128, UInt16}) at boot.jl:708
  Int64(::Ptr) at boot.jl:718
  Int64(::Float32) at float.jl:706
  ...
Stacktrace:
 [1] DateTime(::PyObject, ::Int64, ::Int64, ::Int64, ::Int64, ::Int64, ::Int64, ::Dates.AMPM) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Dates\src\types.jl:367 (repeats 2 times)
 [2] top-level scope at none:1

I've tried other variants of date-related functions but it didn't work.

Datseris avatar Oct 27 '20 10:10 Datseris

@Datseris I've tried to convert a Pandas Dataframe with numpy datetime64[ns] entries into DataFrame from DataFrame.jl.

First I've converted the numpy datetime array to int64: x = x.astype(np.int64)

This will give you the elapsed nanoseconds since 1.1.1970 as integers. In Julia you can convert these integers back into DateTime:

using Dates
x = Nanosecond.(x) + DateTime(1970)

In my case it worked very well.

tjaeuth avatar Mar 05 '21 10:03 tjaeuth

Or as a function, npdt64_to_dt(time) = Nanosecond(time.astype(np.int64)) + DateTime(1970) which can be iterated over the elements of Arrays with dot syntax.

deszoeke avatar Nov 09 '21 21:11 deszoeke