It is personal data and therefore covered by the GDPR. You need to ensure you are acting within the GDPR and so I would suggest having a look at that legislation to check what you should be doing.
I have a copy of the “Guide to the Privacy and Electronic Communications Regulations” where I find that there is no definition of directory, but the regulations refer to:
“a directory of subscribers, whether in printed or electronic form, which is made available to
members of the public or a section of the public, including by means of a directory enquiry service”.
The listing of telephone numbers to identify a caller on a phone’s display is NOT publically available information or service and therefore IMHO not within in the scope of GDPR. Do correct me if I am wrong.