ROS + Caffe 机器人操作系统框架和深度学习框架笔记 (機器人控制與人工智能)

举报
zhangrelay 发表于 2021/07/15 05:05:34 2021/07/15
【摘要】 ROS + Caffe,这里以环境中物体识别为示例,机器人怎么知道环境里面有什么呢? [0.0567392 - n03376595 folding chair][0.0566773 - n04099969 rocking chair, rocker] [0.236507 - n04239074 sliding door] [0.477623 - n03832673...

ROS + Caffe,这里以环境中物体识别为示例,机器人怎么知道环境里面有什么呢?



[0.0567392 - n03376595 folding chair]
[0.0566773 - n04099969 rocking chair, rocker]



[0.236507 - n04239074 sliding door]



[0.477623 - n03832673 notebook, notebook computer]

[0.233582 - n03180011 desktop computer]

caffe在ros中主题以概率方式推断物品,比如椅子,门和笔记本电脑。

如何实现?

首先,需要安装Berkeley Vision and Learning Center (BVLC) 的Caffe,在ubuntu 14.04 16.04上网上教程很多,

这里就不多说,只列出核心步骤:

最重要的就是看官网说明!!!----http://caffe.berkeleyvision.org/

$ sudo apt-get install libatlas-dev libatlas-base-dev

$ sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler

$ sudo apt-get install --no-install-recommends libboost-all-dev

$ sudo apt-get install libboost-all-dev

$ sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev

在github下载最新版caffe:https://github.com/BVLC/caffe

注意库要配置好!可以支持opencv2和opencv3,但是如果配置不正确,呵呵,想象一下,编译和运行错乱是什么结果呢?

简单回答:Segmentation fault (core dump) !用opencv3编译,运行用opencv2,必然挂掉。CUDA ATLAS OpenCV等。

编译比较简单,测试也一般不会出问题。

cp Makefile.config.example Makefile.config
# Adjust Makefile.config (for example, if using Anaconda Python, or if cuDNN is desired)
make all
make test
make runtest

注意,如果只使用CPU,需要修改Makefile.config如下:

cd [CATKIN_WS]/src/ros_caffe/caffe
cp Makefile.config.example Makefile.config
$ vi Makefile.config
For CPU-only Caffe, uncomment CPU_ONLY := 1 in Makefile.config.
$ make all ; make install


编译完成后,测试一下;-) make runtest



ros_caffe示例:https://github.com/tzutalin/ros_caffe

注意caffe库位置:修改Caffe's include and lib 路径 in CMakeLists.txt。

set(CAFFE_INCLUDEDIR caffe/include caffe/install/include)
set(CAFFE_LINK_LIBRARAY caffe/lib)

$ cd [CATKIN_WS]
$ catkin_make
$ source ./devel/setup.bash

出错依据报错信息查找问题。ok,编译工作全部完成。

下载模型之后,直接在分别在终端运行:

$ roscore

$ roslaunch turtlebot_bringup 3dsensor.launch

$ roslaunch astra_launch astra.launch

$ rosrun ros_caffe ros_caffe_test


如果出现:

  

ok,全部完成。可以使用并出现识别结果。

ROS Topics
Publish a topic after classifiction:
/caffe_ret
Receive an image :
/camera/rgb/image_raw




Classifier.h


  
  1. #ifndef CLASSIFIER_H
  2. #define CLASSIFIER_H
  3. #include <iostream>
  4. #include <vector>
  5. #include <sstream>
  6. #include <caffe/caffe.hpp>
  7. #include <opencv2/core/core.hpp>
  8. #include <opencv2/highgui/highgui.hpp>
  9. #include <opencv2/imgproc/imgproc.hpp>
  10. using namespace caffe;
  11. using std::string;
  12. /* Pair (label, confidence) representing a prediction. */
  13. typedef std::pair<string, float> Prediction;
  14. class Classifier {
  15. public:
  16. Classifier(const string& model_file,
  17. const string& trained_file,
  18. const string& mean_file,
  19. const string& label_file);
  20. std::vector<Prediction> Classify(const cv::Mat& img, int N = 5);
  21. private:
  22. void SetMean(const string& mean_file);
  23. std::vector<float> Predict(const cv::Mat& img);
  24. void WrapInputLayer(std::vector<cv::Mat>* input_channels);
  25. void Preprocess(const cv::Mat& img,
  26. std::vector<cv::Mat>* input_channels);
  27. private:
  28. shared_ptr<Net<float> > net_;
  29. cv::Size input_geometry_;
  30. int num_channels_;
  31. cv::Mat mean_;
  32. std::vector<string> labels_;
  33. };
  34. Classifier::Classifier(const string& model_file,
  35. const string& trained_file,
  36. const string& mean_file,
  37. const string& label_file) {
  38. #ifdef CPU_ONLY
  39. Caffe::set_mode(Caffe::CPU);
  40. #else
  41. Caffe::set_mode(Caffe::GPU);
  42. #endif
  43. /* Load the network. */
  44. net_.reset(new Net<float>(model_file, TEST));
  45. net_->CopyTrainedLayersFrom(trained_file);
  46. Blob<float>* input_layer = net_->input_blobs()[0];
  47. num_channels_ = input_layer->channels();
  48. input_geometry_ = cv::Size(input_layer->width(), input_layer->height());
  49. /* Load the binaryproto mean file. */
  50. SetMean(mean_file);
  51. /* Load labels. */
  52. std::ifstream labels(label_file.c_str());
  53. string line;
  54. while (std::getline(labels, line))
  55. labels_.push_back(string(line));
  56. Blob<float>* output_layer = net_->output_blobs()[0];
  57. }
  58. static bool PairCompare(const std::pair<float, int>& lhs,
  59. const std::pair<float, int>& rhs) {
  60. return lhs.first > rhs.first;
  61. }
  62. /* Return the indices of the top N values of vector v. */
  63. static std::vector<int> Argmax(const std::vector<float>& v, int N) {
  64. std::vector<std::pair<float, int> > pairs;
  65. for (size_t i = 0; i < v.size(); ++i)
  66. pairs.push_back(std::make_pair(v[i], i));
  67. std::partial_sort(pairs.begin(), pairs.begin() + N, pairs.end(), PairCompare);
  68. std::vector<int> result;
  69. for (int i = 0; i < N; ++i)
  70. result.push_back(pairs[i].second);
  71. return result;
  72. }
  73. /* Return the top N predictions. */
  74. std::vector<Prediction> Classifier::Classify(const cv::Mat& img, int N) {
  75. std::vector<float> output = Predict(img);
  76. std::vector<int> maxN = Argmax(output, N);
  77. std::vector<Prediction> predictions;
  78. for (int i = 0; i < N; ++i) {
  79. int idx = maxN[i];
  80. predictions.push_back(std::make_pair(labels_[idx], output[idx]));
  81. }
  82. return predictions;
  83. }
  84. /* Load the mean file in binaryproto format. */
  85. void Classifier::SetMean(const string& mean_file) {
  86. BlobProto blob_proto;
  87. ReadProtoFromBinaryFileOrDie(mean_file.c_str(), &blob_proto);
  88. /* Convert from BlobProto to Blob<float> */
  89. Blob<float> mean_blob;
  90. mean_blob.FromProto(blob_proto);
  91. /* The format of the mean file is planar 32-bit float BGR or grayscale. */
  92. std::vector<cv::Mat> channels;
  93. float* data = mean_blob.mutable_cpu_data();
  94. for (int i = 0; i < num_channels_; ++i) {
  95. /* Extract an individual channel. */
  96. cv::Mat channel(mean_blob.height(), mean_blob.width(), CV_32FC1, data);
  97. channels.push_back(channel);
  98. data += mean_blob.height() * mean_blob.width();
  99. }
  100. /* Merge the separate channels into a single image. */
  101. cv::Mat mean;
  102. cv::merge(channels, mean);
  103. /* Compute the global mean pixel value and create a mean image
  104. * filled with this value. */
  105. cv::Scalar channel_mean = cv::mean(mean);
  106. mean_ = cv::Mat(input_geometry_, mean.type(), channel_mean);
  107. }
  108. std::vector<float> Classifier::Predict(const cv::Mat& img) {
  109. Blob<float>* input_layer = net_->input_blobs()[0];
  110. input_layer->Reshape(1, num_channels_,
  111. input_geometry_.height, input_geometry_.width);
  112. /* Forward dimension change to all layers. */
  113. net_->Reshape();
  114. std::vector<cv::Mat> input_channels;
  115. WrapInputLayer(&input_channels);
  116. Preprocess(img, &input_channels);
  117. net_->ForwardPrefilled();
  118. /* Copy the output layer to a std::vector */
  119. Blob<float>* output_layer = net_->output_blobs()[0];
  120. const float* begin = output_layer->cpu_data();
  121. const float* end = begin + output_layer->channels();
  122. return std::vector<float>(begin, end);
  123. }
  124. /* Wrap the input layer of the network in separate cv::Mat objects
  125. * (one per channel). This way we save one memcpy operation and we
  126. * don't need to rely on cudaMemcpy2D. The last preprocessing
  127. * operation will write the separate channels directly to the input
  128. * layer. */
  129. void Classifier::WrapInputLayer(std::vector<cv::Mat>* input_channels) {
  130. Blob<float>* input_layer = net_->input_blobs()[0];
  131. int width = input_layer->width();
  132. int height = input_layer->height();
  133. float* input_data = input_layer->mutable_cpu_data();
  134. for (int i = 0; i < input_layer->channels(); ++i) {
  135. cv::Mat channel(height, width, CV_32FC1, input_data);
  136. input_channels->push_back(channel);
  137. input_data += width * height;
  138. }
  139. }
  140. void Classifier::Preprocess(const cv::Mat& img,
  141. std::vector<cv::Mat>* input_channels) {
  142. /* Convert the input image to the input image format of the network. */
  143. cv::Mat sample;
  144. if (img.channels() == 3 && num_channels_ == 1)
  145. cv::cvtColor(img, sample, CV_BGR2GRAY);
  146. else if (img.channels() == 4 && num_channels_ == 1)
  147. cv::cvtColor(img, sample, CV_BGRA2GRAY);
  148. else if (img.channels() == 4 && num_channels_ == 3)
  149. cv::cvtColor(img, sample, CV_BGRA2BGR);
  150. else if (img.channels() == 1 && num_channels_ == 3)
  151. cv::cvtColor(img, sample, CV_GRAY2BGR);
  152. else
  153. sample = img;
  154. cv::Mat sample_resized;
  155. if (sample.size() != input_geometry_)
  156. cv::resize(sample, sample_resized, input_geometry_);
  157. else
  158. sample_resized = sample;
  159. cv::Mat sample_float;
  160. if (num_channels_ == 3)
  161. sample_resized.convertTo(sample_float, CV_32FC3);
  162. else
  163. sample_resized.convertTo(sample_float, CV_32FC1);
  164. cv::Mat sample_normalized;
  165. cv::subtract(sample_float, mean_, sample_normalized);
  166. /* This operation will write the separate BGR planes directly to the
  167. * input layer of the network because it is wrapped by the cv::Mat
  168. * objects in input_channels. */
  169. cv::split(sample_normalized, *input_channels);
  170. }
  171. #endif



  
  1. #include <ros/ros.h>
  2. #include <ros/package.h>
  3. #include <std_msgs/String.h>
  4. #include <image_transport/image_transport.h>
  5. #include <cv_bridge/cv_bridge.h>
  6. #include "Classifier.h"
  7. const std::string RECEIVE_IMG_TOPIC_NAME = "camera/rgb/image_raw";
  8. const std::string PUBLISH_RET_TOPIC_NAME = "/caffe_ret";
  9. Classifier* classifier;
  10. std::string model_path;
  11. std::string weights_path;
  12. std::string mean_file;
  13. std::string label_file;
  14. std::string image_path;
  15. ros::Publisher gPublisher;
  16. void publishRet(const std::vector<Prediction>& predictions);
  17. void imageCallback(const sensor_msgs::ImageConstPtr& msg) {
  18. try {
  19. cv_bridge::CvImagePtr cv_ptr = cv_bridge::toCvCopy(msg, "bgr8");
  20. //cv::imwrite("rgb.png", cv_ptr->image);
  21. cv::Mat img = cv_ptr->image;
  22. std::vector<Prediction> predictions = classifier->Classify(img);
  23. publishRet(predictions);
  24. } catch (cv_bridge::Exception& e) {
  25. ROS_ERROR("Could not convert from '%s' to 'bgr8'.", msg->encoding.c_str());
  26. }
  27. }
  28. // TODO: Define a msg or create a service
  29. // Try to receive : $rostopic echo /caffe_ret
  30. void publishRet(const std::vector<Prediction>& predictions) {
  31. std_msgs::String msg;
  32. std::stringstream ss;
  33. for (size_t i = 0; i < predictions.size(); ++i) {
  34. Prediction p = predictions[i];
  35. ss << "[" << p.second << " - " << p.first << "]" << std::endl;
  36. }
  37. msg.data = ss.str();
  38. gPublisher.publish(msg);
  39. }
  40. int main(int argc, char **argv) {
  41. ros::init(argc, argv, "ros_caffe_test");
  42. ros::NodeHandle nh;
  43. image_transport::ImageTransport it(nh);
  44. // To receive an image from the topic, PUBLISH_RET_TOPIC_NAME
  45. image_transport::Subscriber sub = it.subscribe(RECEIVE_IMG_TOPIC_NAME, 1, imageCallback);
  46. gPublisher = nh.advertise<std_msgs::String>(PUBLISH_RET_TOPIC_NAME, 100);
  47. const std::string ROOT_SAMPLE = ros::package::getPath("ros_caffe");
  48. model_path = ROOT_SAMPLE + "/data/deploy.prototxt";
  49. weights_path = ROOT_SAMPLE + "/data/bvlc_reference_caffenet.caffemodel";
  50. mean_file = ROOT_SAMPLE + "/data/imagenet_mean.binaryproto";
  51. label_file = ROOT_SAMPLE + "/data/synset_words.txt";
  52. image_path = ROOT_SAMPLE + "/data/cat.jpg";
  53. classifier = new Classifier(model_path, weights_path, mean_file, label_file);
  54. // Test data/cat.jpg
  55. cv::Mat img = cv::imread(image_path, -1);
  56. std::vector<Prediction> predictions = classifier->Classify(img);
  57. /* Print the top N predictions. */
  58. std::cout << "Test default image under /data/cat.jpg" << std::endl;
  59. for (size_t i = 0; i < predictions.size(); ++i) {
  60. Prediction p = predictions[i];
  61. std::cout << std::fixed << std::setprecision(4) << p.second << " - \"" << p.first << "\"" << std::endl;
  62. }
  63. publishRet(predictions);
  64. ros::spin();
  65. delete classifier;
  66. ros::shutdown();
  67. return 0;
  68. }

问题汇总:一般哪有那么顺利~~


  
  1. I0122 16:50:22.870820 20921 upgrade_proto.cpp:44] Attempting to upgrade input file specified using deprecated transformation parameters: /home/exbot/Rob_Soft/caffe/src/ros_caffe-master/data/bvlc_reference_caffenet.caffemodel
  2. I0122 16:50:22.870877 20921 upgrade_proto.cpp:47] Successfully upgraded file specified using deprecated data transformation parameters.
  3. W0122 16:50:22.870887 20921 upgrade_proto.cpp:49] Note that future Caffe releases will only support transform_param messages for transformation fields.
  4. I0122 16:50:22.870893 20921 upgrade_proto.cpp:53] Attempting to upgrade input file specified using deprecated V1LayerParameter: /home/exbot/Rob_Soft/caffe/src/ros_caffe-master/data/bvlc_reference_caffenet.caffemodel
  5. I0122 16:50:23.219712 20921 upgrade_proto.cpp:61] Successfully upgraded file specified using deprecated V1LayerParameter
  6. I0122 16:50:23.221462 20921 net.cpp:746] Ignoring source layer data
  7. I0122 16:50:23.316156 20921 net.cpp:746] Ignoring source layer loss
  8. Segmentation fault (core dumped)

$ ulimit -a

$ ulimit -c unlimited

$ echo 1 | sudo tee /proc/sys/kernel/core_uses_pid

如果查看出错信息:

$ gdb ros_caffe_test core.20921

GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ros_caffe_test...(no debugging symbols found)...done.
[New LWP 20921]
[New LWP 20934]
[New LWP 20933]
[New LWP 20939]
[New LWP 20932]
[New LWP 20930]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/exbot/Rob_Soft/caffe/devel/lib/ros_caffe/ros_caffe_test'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fc51da27e0c in cv::merge(cv::_InputArray const&, cv::_OutputArray const&) ()
   from /usr/lib/x86_64-linux-gnu/libopencv_core.so.2.4
(gdb) 

基本推断,是opencv库乱了,调整环境变量指定。


文章来源: zhangrelay.blog.csdn.net,作者:zhangrelay,版权归原作者所有,如需转载,请联系作者。

原文链接:zhangrelay.blog.csdn.net/article/details/54669922

【版权声明】本文为华为云社区用户转载文章,如果您发现本社区中有涉嫌抄袭的内容,欢迎发送邮件进行举报,并提供相关证据,一经查实,本社区将立刻删除涉嫌侵权内容,举报邮箱: cloudbbs@huaweicloud.com
  • 点赞
  • 收藏
  • 关注作者

评论(0

0/1000
抱歉,系统识别当前为高风险访问,暂不支持该操作

全部回复

上滑加载中

设置昵称

在此一键设置昵称,即可参与社区互动!

*长度不超过10个汉字或20个英文字符,设置后3个月内不可修改。

*长度不超过10个汉字或20个英文字符,设置后3个月内不可修改。