Context in the United StatesIn the United States colleges and universities are often referred to interchangeably with little meaning or weight being given to one or the other. That said this isn't always the case so it can get a bit tricky.
One distinction that exists is that often universities are larger institutions with many more programs offered, including graduate degrees. Colleges can often have a more limited scope of programs offered. Universities may also consider different faculties within the institution to be considered a college. For example a University may refer to its college of medicine or its law college.
In conversation most in the United States will refer to "going to college" even if the name of their school is actually a university.
In the United States the distinction between a college and a university really depends on the individual qualifications of the school as there are prestigious institutions that use both names.
UK, Canada, and AustraliaOutside of the United States in many countries (specifically the UK, Canada, and Australia) there is a clear distinction between colleges and universities.
Universities are universally the major institutions that offer a wide variety of programs and degrees ranging from undergraduate degrees to doctorates. Colleges on the other hand tend to smaller institutions that offer very limited scopes of programs ending at undergraduate degrees. In many cases colleges will also be essentially preparatory schools that students attend before moving on to university for their final degree. Colleges are often also considered technical schools that offer programs focused on preparing people for work in trades. There is also typically a very clear distinction between the deemed quality of degrees awarded from colleges and universities, with university degrees being considered more prestigious. In discussion people in these countries will say they're "going to university" in a conversational context if they are actually attending a university.
These conversational differences can actually drive some inter-cultural misunderstanding. If an American student tells a Canadian student they are going to college, there will be an initial impression that the American student goes to a second tier school, regardless of their actual institution.