This article on infoq talks about the story point and its relation to complexity or time. The argument seems to be it should be only related to effort and not complexity
When team members say Extra large for a story, generally it can mean one of the following.
a) I don’t have all the information needed and hence I think it is large ( This is not complexity or effort. This is the ambiguity or uncertainty. In this case, we cannot really base the size on Effort as effort can only come to play when the team member knows that he or she needs to do
b) I know all about the story but this is complex. This is a good case to say this is complexity or more so effort. Its going to take me 10 days to do it as an example.
Story sizing is all about the relative measure.
The real effort only comes to play when you work on a story. At times what is a small story becomes extra-large, the effort grows. So there should not be a correlation to time alone. If we did time alone then why do we have to do task breakdowns?
Estimate size and derive duration. This picture captures the intent.